Three-dimensional (3D) graphics processing and display are powerful technologies for capturing and retaining a person's attention. This is because 3D video productions and holographic displays capture a degree of realism or invoke greater perceived familiarity that a two-dimensional (2D) production or display simply cannot match. A prime application of 3D processing is in the areas of augmented reality (AR) and augmented virtuality (AV). These two different terms generally describe approaches to integrating information or context from a real environment with a virtual (or computer generated) one, but in AR the real elements predominate while in AV the virtual ones predominate. Similar to AR, many special effects in television programs and movies incorporate virtual images with computer generated ones. Alternatively, primarily animated programs that occasionally incorporate images, video, or the motions (e.g., animating a computer generated character based on the observed motions of an actor) of real actors or scenes are closer to AV. AR and AV differ from movies and most television in that they are typically performed in real-time with live participants.
The term participant is used in the present disclosure because AR and AV applications are generally interactive in nature such that a person captured by the camera or cameras of such a system is invited to participate in some activity or experience. The real-time interface allows a participant to interact with a virtual scene or virtual 3D object and see those interactions as they happen. For the purposes of the present disclosure, the term AR will be used inclusively to refer to augmented reality and augmented virtuality applications.
Existing techniques for capturing information about a participant and the participant's environment and then registering that information in 3D typically require specialized hardware components and/or complex setups of hardware components. The term registration is used to describe direct capture or inference of the 3D position and volume of a participant. The registration process is analogous to an architect adding a realistic model of an existing building to a 3D model of a downtown block to show a model of a planned building design in the context of its intended location. In this analogy, the entire model represents the virtual 3D scene with 3D representations of existing real buildings and a 3D representation of a purely virtual building. Registration in the context of AR includes direct measurement of a participant and entry of the participant's measurements, and also includes inferences based on one or more observations of the participant.
In one example of a system for registering a participant in a virtual 3D scene, two or more cameras may be used in a carefully positioned arrangement (e.g., a stereoscopic camera rig) to enable computerized depth perception. Alternatively, a specialized time-of-flight camera provides a similar level of depth perception using special sensor and/or lighting elements. In yet another setup, a laser range-finder or radar setup may provide this depth information to augment video captured by a camera. Finally, additional information may be gathered from the scene by projecting (typically intermittently) a uniform grid across the camera's field of view (much like a high end digital camera auto focuses its lens) or by using fiducial markers (easily recognized reference markers) attached to the participant. The latter technique is used in medical imaging and studio production of movies and television.
Each of these techniques has limitations such as a requirement for specialized hardware or complex arrangements of hardware, significant computer processing requirements that make real-time interaction expensive and difficult, and/or require a participant to first attach tags or markers to their body and/or clothing before participating in the AR environment.